Defect detection of a component in an assembly

ABSTRACT

A system for validating installation correctness of a component in a test assembly includes a housing having a platform including a tiered surface. The tiered surface forms an abutment surface configured as a stop against which a test assembly is abutted. A plurality of cameras is positioned to capture different views of the test assembly. A processing device is configured to execute instructions to capture an image from each of the plurality of cameras of the test assembly that includes a plurality of components. Each of the plurality of components is analyzed within each image of the plurality of images. A matching score is determined and an indication of whether the plurality of components was correctly installed in the test assembly is generated.

FIELD OF THE TECHNOLOGY

At least some embodiments disclosed herein relate generally to componentinstallation validation. More particularly, the embodiments relate toartificial intelligence systems and methods for computer-aidedvalidation of the installation of a component in an assembly such as acomputing device.

BACKGROUND

Device manufacturing, such as mobile devices (e.g., smartphones,tablets, smartwatches, or the like) utilize several components assembledtogether. The assembly process can include, for example, securing thecomponents together via fasteners (e.g., screws or the like). Theassembly process, if not completed properly (e.g., missing screws,incorrect screws, improperly tightened screws, or the like), can causequality control issues.

Improved methods and systems for validating the proper installation ofthe various components are desirable.

BRIEF DESCRIPTION OF THE DRAWINGS

References are made to the accompanying drawings that form a part ofthis disclosure and illustrate embodiments in which the systems andmethods described in this Specification can be practiced.

FIG. 1 shows a system for validation of installation of a component inan assembly, according to an embodiment.

FIG. 2 shows a portion of the system for validation of installation of acomponent of an assembly of FIG. 1, according to an embodiment.

FIG. 3 shows a schematic architecture for the system of FIG. 1,according to an embodiment.

FIG. 4 shows a flowchart for a method of validating installation of acomponent in an assembly, according to an embodiment.

FIG. 5 shows a first graphical user interface (GUI), according to anembodiment.

FIG. 6 shows a second GUI, according to an embodiment.

FIG. 7 shows a third GUI, according to an embodiment.

Like reference numbers represent like parts throughout.

DETAILED DESCRIPTION

Computing devices such as, but not limited to, smartphones, tablets,laptops, smartwatches, and the like, include numerous components thatare assembled together. The assembly process can include fasteners(e.g., screws or the like) that keep the various components secured. Itis important that these fasteners be installed correctly (e.g., allscrews installed (e.g., no missing screws), proper screws installed,screws properly tightened, or the like) as part of the quality controlprocess.

The embodiments disclosed herein are directed to a system and method forinspecting components (e.g., fasteners such as screws or the like) of anassembly (e.g., a computing device such as, but not limited to, asmartphone, a tablet, a laptop, a smartwatch, a cellphone, or the like)during the manufacturing process. The system includes a platform thatholds the test assembly (i.e., the assembly being inspected) in adefined position. The defined position is established relative to aplurality of cameras that are arranged to capture different views of theassembly.

When an assembly (e.g., a computing device) is placed onto the platform,an image of the assembly is captured from each of the plurality ofcameras. The plurality of cameras are triggered to capture each of theimages at the different viewing angles in response to a combination ofproximity and motion sensors that identify when the assembly is inplace. “Profile images” (i.e., images of a model assembly (e.g., a modelassembly (i.e., screws properly installed)) can be captured in acalibration process. The platform maintains the assembly in a generallyfixed location. As a result, each image of an assembly being validated(i.e., a test assembly) is taken in the same coordinate system or with apredetermined relationship maintained by the platform relative to thecorresponding profile image.

Each of the captured images is compared against the correspondingprofile image to determine a matching score representative of whetherthe components were correctly installed.

In an embodiment, the different viewing angles can provide a moreaccurate response, as the installation defect may be noticeable in oneof the views but appear correct (or not be noticeable) in another of theviews. The matching score is based on the comparison of each view. Inresponse to the determination, an indicator (correct, incorrect,requires further checking) including one of the profile images is thendisplayed on an interface of the system.

A system is disclosed. The system includes a housing, enclosing aplatform having a tiered surface. The tiered surface includes a firstsurface and a second surface parallel to the first surface and disposeda height from the first surface. The tiered surface forms an abutmentsurface configured as a stop against which a test assembly is abutted. Aplurality of cameras are enclosed in the housing, each of the pluralityof cameras positioned to capture a different view of the test assembly.A processing device is configured to execute instructions to capture animage from each of the plurality of cameras of the test assembly thatincludes a plurality of components. Each of the plurality of componentswithin each image of the plurality of images is analyzed. A matchingscore is determined between a corresponding profile image of each of theplurality of components within each of the plurality of images of thetest assembly. The installation of each of the plurality of componentsis classified based on the matching score. An indication of whether theplurality of components was correctly installed in the test assembly isgenerated based on the matching score. A display device is configured todisplay a graphical user interface (GUI) based on the indication asgenerated.

A method for validating correctness of installation of a component in anassembly is also disclosed. The method includes capturing a plurality ofimages using a plurality of cameras. The plurality of cameras areoriented to capture different views of a test assembly including acomponent. A processing device analyzes the component within each imageof the plurality of images. The processing device determines a matchingscore between a profile image of the component within each correspondingprofile image and the plurality of captured images. The componentinstallation is classified based on the matching score. An indication ofwhether the component was correctly installed in the test assembly basedon the matching score is generated.

FIG. 1 shows a system 10 for validation of installation of a componentin an assembly 15, according to an embodiment. The system 10 cangenerally be used to, for example, validate whether a fastener (e.g., ascrew or the like) or other component is properly installed in theassembly 15. In an embodiment, the validation can be part of a qualitycontrol process during manufacturing. In an embodiment, the system 10can be used to validate whether the fastener or other component isproperly installed at a point after manufacturing (e.g., duringrefabricating, maintenance, or the like).

In the illustrated embodiment, the assembly 15 is a smartphone. It is tobe appreciated that the smartphone is an example, and the assembly 15can vary beyond a smartphone. Examples of other assemblies 15 include,but are not limited to, a tablet, a smartwatch, a mobile phone otherthan a smartphone, a personal digital assistant (PDA), a laptopcomputing device, or the like. Furthermore, the maker or manufacturer ofthe assembly 15 is not limited. That is, the system 10 can be used tovalidate the installation correctness of components in assemblies 15from different manufacturers so long as a calibration procedure isperformed to create a profile image for the corresponding assembly 15.

The system 10 includes a display 20 for displaying results of thevalidation to the user. In an embodiment, the display 20 can be acombined display and input (e.g., a touchscreen). In an embodiment, thedisplay 20 can be a display of a tablet or the like. In such anembodiment, a memory of the tablet can store one or more programs to beexecuted by a processing device of the tablet for validating thecorrectness of the installation of the component in the assembly 15.

In the illustrated embodiment, the display 20 is secured to housing 25of the system 10. In an embodiment, the display 20 can be separate fromthe housing 25 (i.e., not secured to the housing 25, but positioned nearthe system 10 and electronically connected to the system 10). However,it may be beneficial to secure the display 20 to the housing 25 toreduce a footprint of the system 10.

A platform 30 is utilized to position the assembly 15 within the system10 for validation. The platform 30 enables each assembly 15 placed intothe system 10 for validation to be placed in substantially the samelocation. As a result, an amount of effort in determining whether theprofile image and the assembly 15 under test (test assembly) is in asame location relative to cameras of the system 10 can be reduced. Theplatform 30 is shown and described in additional detail in accordancewith FIG. 2 below.

In an embodiment, the system 10 can be portable. For example, theillustrated embodiment shows system 10 with a handle 35 for carrying thesystem 10. It is to be appreciated that portability of the system 10 isoptional, and accordingly, the handle 35 is optional. In an embodiment,the system 10 may be sized differently based on the type of assembly 15to be validated.

FIG. 2 shows the platform 30 of the system 10 of FIG. 1 for validationof installation of a component in an assembly 15, according to anembodiment.

The platform 30 includes a tiered surface having a first surface 40 anda second surface 45. A step is thus formed between the first surface 40and the second surface 45. A plane of the first surface 40 and a planeof the second surface 45 are parallel. In the illustrated embodiment,the second surface 40 is L-shaped when viewed from a top view.

The second surface 45 is positioned a height H from the first surface40. The height H between the first surface 40 and the second surface 45creates an abutment surface 50.

The height H is selected such that the abutment surface 50 serves as astop for the assembly 15 when placed within the system 10. The abutmentsurface 50 is configured to provide a stop for the assembly 15 on twosides of the assembly 15 (i.e., a major dimension of the assembly 15 anda minor dimension of the assembly 15).

The height H is selected to be smaller than a thickness T of theassembly 15 being validated in the system 10. The height H is selectedto be smaller than the thickness T of the assembly 15 to not hinder sideviews of the assembly 15. The height H is selected to be large enoughthat an operator inserting the assembly 15 can abut the assembly 15 withthe abutment surface 50. In this manner, the abutment surface 50 servesas a stop for the operator when inserting the assembly 15 into thesystem 10. In an embodiment, the height H can be substantially the sameas the thickness T of the assembly 15.

The configuration of the platform 30 is helpful in establishing thelocation of the assembly 15. By including the platform 30, the system 10can be calibrated to generate the profile images using a single assemblysince the coordinate system is generally fixed. The platform 30 can, asa result, be used to account for minor variations in placement of theassembly 15 by the operator as the offset from the expected coordinatedsystem can be determined based on the location of the assembly 15relative to a calibration assembly 15.

FIG. 3 shows a schematic architecture for the system 10 of FIG. 1,according to an embodiment.

The system 10 generally includes a plurality of cameras 100; a motionsensor 105; a proximity sensor 110; a processing device 115, memory 120,a network input/output (I/O) 125, user I/O 130, storage 135, and aninterconnect 140. The processing device 115, memory 120, networkinput/output (I/O) 125, user I/O 130, storage 135, and interconnect 140can be within the housing 25 in an embodiment. In an embodiment, theprocessing device 115, memory 120, network input/output (I/O) 125, userI/O 130, storage 135, and interconnect 140 can be external from thehousing 25.

The plurality of cameras 100 are arranged in the system 10 to capturedifferent views of the assembly 15. In an embodiment, the cameras 100are digital cameras. For example, in an embodiment the system 10includes three cameras 100 arranged to capture a top view, an up-frontview, and an up-side view. In an embodiment, the system 10 includes fourcameras 100 arranged to capture a top view, an up-front view, a firstup-side view, and a second (opposite) up-side view. It will beappreciated that a single camera 100 could be used, although accuracymay be improved when a plurality of cameras 100 are used as a componentmay appear to be correctly installed in a first view but be determinedto be incorrectly installed in a second view.

The motion sensor 105 can be, for example, a laser sensor that can betriggered when an object (i.e., assembly 15) breaks the laser signal.The motion sensor 105 can be installed at the opening to the housing 25.In an embodiment, the motion sensor 105 may not be included.

The proximity sensor 110 can be a sensor to determine when an object isplaced near it. The proximity sensor 110 can be placed in the platform30 of the system 10. In an embodiment, when the motion sensor 105 istriggered and the proximity sensor 110 detects an object, the cameras100 can capture images of the assembly 15 on the platform 30. In anembodiment, the proximity sensor 110 can be included regardless ofwhether the motion sensor 105 is present. In an embodiment with bothmotion sensor 105 and proximity sensor 110, the image capturing may beperformed after the proximity sensor 110 detects the assembly 15.

In an embodiment, automatically causing the image capturing andsubsequent validation to be performed using the proximity sensor 110, ora combination of the proximity sensor 110 and the motion sensor 105, canincrease a number of assemblies 15 that can be validated in a setperiod. That is, reducing effort of a human operator, or even allowingfor a robotic arm to load the assembly 15 into the system 10 forvalidation, can reduce an amount of time and effort needed to review thequality of the manufacturing process.

The processing device 115 can retrieve and execute programminginstructions stored in the memory 120, the storage 135, or combinationsthereof. The processing device 115 can also store and retrieveapplication data residing in the memory 120. The programminginstructions can perform the method described in accordance with FIG. 4below to determine whether the components of the assembly 15 areproperly installed, and additionally, cause display of one of thegraphical user interfaces (GUIs) shown and described in accordance withFIGS. 5-7 below.

The interconnect 140 is used to transmit programming instructions and/orapplication data between the processing device 115, the user I/O 130,the memory 120, the storage 135, and the network I/O 125. Theinterconnect 140 can, for example, be one or more busses or the like.The processing device 115 can be a single processing device, multipleprocessing devices, or a single processing device having multipleprocessing cores. In an embodiment, the processing device 115 can be asingle-threaded processing device. In an embodiment, the processingdevice 115 can be a multi-threaded processing device.

The memory 120 is generally included to be representative of arandom-access memory such as, but not limited to, Static Random-AccessMemory (SRAM), Dynamic Random-Access Memory (DRAM), or Flash. In anembodiment, the memory 120 can be a volatile memory. In an embodiment,the memory 120 can be a non-volatile memory. In an embodiment, at leasta portion of the memory 120 can be virtual memory.

The storage 135 is generally included to be representative of anon-volatile memory such as, but not limited to, a hard disk drive, asolid-state device, removable memory cards, optical storage, flashmemory devices, network attached storage (NAS), or connections tostorage area network (SAN) devices, or other similar devices that maystore non-volatile data. In an embodiment, the storage 135 is a computerreadable medium. In an embodiment, the storage 135 can include storagethat is external to the user device, such as in a cloud.

FIG. 4 shows a flowchart for a method 200 of validating installation ofa component in a test assembly 15 using artificial intelligence,according to an embodiment.

At block 205 a test assembly 15 is loaded into the system 10. Thisincludes abutting the assembly 15 with the abutment surface 50 of theplatform 30. In an embodiment, the test assembly 15 can be loaded by ahuman operator. In an embodiment, a robotic or mechanical arm can beautomated to place the test assembly 15 onto the platform 30. Theplacement of the test assembly 15 can cause the motion sensor 105, theproximity sensor 110, or a combination thereof, to generate a signalindicative of the assembly 15 being in place.

At block 210, in response to the signal generated by the motion sensor105, the proximity sensor 110, or a combination thereof, the pluralityof cameras 100 each capture an image. As discussed above, the cameras100 are oriented such that the captured images are of different views ofthe test assembly 15.

At block 215, each component within each of the captured images iscompared against the corresponding component in the correspondingprofile image. In an embodiment, the comparison can include, for eachcomponent location, comparing multiple portions of the captured imagearound the component. For example, if a center of the component is apoint (x, y) in the profile image, a first portion would be at (x, y),another at (x+1, y+1), another at (x−1, y−1), etc. In this manner, amatching score can be computed for each portion of the captured imagerelative to the profile image. In an embodiment, the matching can becomputed using a similarity metric such as, but not limited to,correlation coefficient score. A sampling of 50-500 points around (x, y)can be performed to provide a higher accuracy in making the validation.This improvement in accuracy can, for example, be because the multiplesamples can compensate for minor displacements (e.g., 0-20 pixels or 1-2mm in physical space) of the components being tested relative to thecomponents in the profile image. Thus, the sampling can account forminor variations in the placement of the assembly 15 at block 205.

The highest matching score among the portions of the captured image canbe considered the score for the component being examined. It is to beappreciated that other statistical values could be used for the matchingscore, such as, but not limited to, the mean, median, or mode.

In performing the similarity comparison, several bases for determining acomponent is incorrect can be captured. For example, the validation canidentify missing components, improperly installed components (e.g.,wrong screw or the like), or installations that are not complete (e.g.,a screw is not tightened or is over-tightened) based on orientation ofthe component. It is to be appreciated that the matching score will beimpacted more heavily by a missing or incorrect component than by amisoriented component. The lowest matching score for each component iscompared across the plurality of images and is retained as the matchingscore for that component.

At block 220 an output is generated indicative of the results of thevalidation (e.g., correct, incorrect, needs checking). The output can bebased on a range of the matching score. That is, if the matching scoreis greater than a first value, then the output can be that theinstallation is correct; between the first value and a lower secondvalue, the installation may need checking; and between the lower secondvalue and a third value that is lower than the second value, theinstallation may be incorrect.

At block 225, the output is displayed on the display 20 of the system10. In an embodiment, the display can include a notation of whichcomponents are incorrect or need checking. In an embodiment, to reduceprocessing effort and increase a speed of the output, the display caninclude one of the profile images (instead of one of the captured imagesfor the test assembly 15).

FIG. 5 shows a first graphical user interface (GUI) 250, according to anembodiment. The first GUI 250 is generally representative of a GUIdisplayed when the manufacturing component was validated and determinedto be correctly installed.

FIG. 6 shows a second GUI 300, according to an embodiment. The secondGUI 300 is generally representative of a GUI displayed when themanufacturing component was incorrectly installed.

FIG. 7 shows a third GUI 350, according to an embodiment. The third GUI350 is generally representative of a GUI displayed when themanufacturing component was determined to have a potential issue, or anappropriate confidence factor was not met.

The GUIs 250-350 in FIGS. 5-7 generally include an indicator 255. In theGUIs 300 and 350, because the GUIs are representative of situationswhere the installation of the component was not validated as beingcorrect, a profile image 260 is displayed showing the correct assembly.It is to be appreciated that the GUI 250 could also display the profileimage 260. The indicator 255 can vary based on the GUI to visuallyrepresent to the operator whether the component was correctly installed(GUI 250), incorrectly installed (GUI 300), or needs checking (GUI 350).The indicators 255 in the illustrated embodiments are representative andare not intended to be limiting. The image 260 in each of the GUIs 300and 350 is the same and is a top view of the profile image. It is to beappreciated that the image 260 can vary. That is, a different view canbe shown in an embodiment.

Examples of computer-readable storage media include, but are not limitedto, any tangible medium capable of storing a computer program for use bya programmable processing device to perform functions described hereinby operating on input data and generating an output. A computer programis a set of instructions that can be used, directly or indirectly, in acomputer system to perform a certain function or determine a certainresult. Examples of computer-readable storage media include, but are notlimited to, a floppy disk; a hard disk; a random access memory (RAM); aread-only memory (ROM); a semiconductor memory device such as, but notlimited to, an erasable programmable read-only memory (EPROM), anelectrically erasable programmable read-only memory (EEPROM), Flashmemory, or the like; a portable compact disk read-only memory (CD-ROM);an optical storage device; a magnetic storage device; other similardevice; or suitable combinations of the foregoing.

In some embodiments, hardwired circuitry may be used in combination withsoftware instructions. Thus, the description is not limited to anyspecific combination of hardware circuitry and software instructions,nor to any source for the instructions executed by the data processingsystem.

The terminology used herein is intended to describe embodiments and isnot intended to be limiting. The terms “a,” “an,” and “the” include theplural forms as well, unless clearly indicated otherwise. The terms“comprises” and/or “comprising,” when used in this Specification,specify the presence of the stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, and/or components.

It is to be understood that changes may be made in detail, especially inmatters of the construction materials employed and the shape, size, andarrangement of parts without departing from the scope of the presentdisclosure. This Specification and the embodiments described areexamples, with the true scope and spirit of the disclosure beingindicated by the claims that follow.

What is claimed is:
 1. A system, comprising: a housing, enclosing: aplatform having a tiered surface, comprising a first surface and asecond surface parallel to the first surface and disposed a height fromthe first surface, wherein the tiered surface forms an abutment surfaceconfigured as a stop against which a test assembly is abutted; and aplurality of cameras, each of the plurality of cameras positioned tocapture a different view of the test assembly; a processing device, theprocessing device configured to execute instructions to: capture animage from each of the plurality of cameras of the test assembly thatincludes a plurality of components; analyze each of the plurality ofcomponents within each image of the plurality of images; determine amatching score between a corresponding profile image of each of theplurality of components within each of the plurality of images of thetest assembly; classify the installation of each of the plurality ofcomponents based on the matching score; and generate an indication ofwhether the plurality of components was correctly installed in the testassembly based on the matching score; and a display device configured todisplay a graphical user interface (GUI) based on the indication asgenerated.
 2. The system of claim 1, comprising a motion sensor on thehousing.
 3. The system of claim 1, comprising a proximity sensor on theplatform.
 4. The system of claim 3, wherein the processing device isconfigured to automatically capture the plurality of images in responseto an output from the proximity sensor.
 5. The system of claim 1,wherein the plurality of cameras includes three cameras oriented tocapture a top view, an up-front view, and an up-side view.
 6. The systemof claim 1, wherein the display device is secured to the housing.
 7. Thesystem of claim 1, wherein the processing device is configured to causethe display device to display the indication of whether the componentwas correctly installed in the assembly based on the matching score. 8.The system of claim 1, wherein the second surface is L-shaped whenviewed from a top view.
 9. The system of claim 1, wherein the processingdevice and the display device are integrated into a single device.
 10. Amethod for validating correctness of installation of a component in adevice, comprising: capturing a plurality of images using a plurality ofcameras, wherein the plurality of cameras are oriented to capture adifferent view of a test assembly including a component; analyzing, by aprocessing device, the component within each image of the plurality ofimages; determining, by the processing device, a matching score betweena profile image of the component within each corresponding profile imageand the plurality of captured images; classifying the componentinstallation based on the matching score; and generating an indicationof whether the component was correctly installed in the test assemblybased on the matching score.
 11. The method of claim 10, wherein thecomponent is a screw and the test assembly is a smartphone.
 12. Themethod of claim 10, wherein the classifying includes one of a correctinstallation, a defective installation, or a further inspection neededclassification.
 13. The method of claim 10, wherein the matching scoreis determined to be a largest matching score of the component within theplurality of captured images.
 14. The method of claim 10, wherein thecapturing the plurality of images using the plurality of cameras istriggered in response to a proximity sensor detecting the test assembly.15. The method of claim 10, comprising displaying a GUI based on theindication as generated.
 16. The method of claim 15, wherein the GUIincludes an indicator and one of the plurality of profile images. 17.The method of claim 10, wherein the indication as generated includes adescription of which of the components was a defective installation orneeds further inspection.
 18. The method of claim 10, wherein theanalyzing the component within each image of the plurality of imagesincludes comparing each component with the profile image to determine acorrelation coefficient score.
 19. The method of claim 10, wherein theanalyzing the component within each image of the plurality of imagesincludes comparing a plurality of portions of the image with the profileimage.
 20. The method of claim 10, wherein the test assembly includes aplurality of components, each of the plurality of components beinganalyzed.